3 Long Short Term

\Recurrent backprop" for learning to store information over extended time periods takes too long. The main reason is insuucient, decaying error back ow. We describe a novel, ee-cient \Long Short Term Memory" (LSTM) that overcomes this and related problems. Unlike previous approaches, LSTM can learn to bridge arbitrary time lags by enforcing constant error ow. Using gradient descent, LSTM explicitly learns when to store information and when to access it. In experimental comparisons with \Real-Time Recurrent Learning", \Recurrent Cascade-Correlation", \Elman nets", and \Neural Sequence Chunking", LSTM leads to many more successful runs, and learns much faster. Unlike its competitors, LSTM can solve tasks involving minimal time lags of more than 1000 time steps, even in noisy environments.